This PR implements comprehensive queue monitoring for Zeno's processing pipeline, allowing operators to identify bottlenecks by tracking the queue sizes of each component through Prometheus metrics.

Overview

The implementation adds real-time monitoring of all pipeline channels with both Prometheus and TUI integration. This enables operators to see which components are slowing down the crawl by having the largest queues.

Key Features

Pipeline Component Tracking

All 8 pipeline components are now monitored with descriptive names:

• reactor_to_preprocessor - Items flowing from reactor to preprocessor
• preprocessor_to_archiver - Items flowing from preprocessor to archiver
• archiver_to_postprocessor - Items flowing from archiver to postprocessor
• postprocessor_to_finisher - Items flowing from postprocessor to finisher
• finisher_to_source - Finished items returned to source (HQ/LQ)
• source_to_finisher - New items from source to finisher
• hq_outlinks_finish - HQ outlinks finish channel
• hq_outlinks_produce - HQ outlinks produce channel

Prometheus Metrics Integration

Component queue sizes are exposed as Prometheus metrics with the following format:

# HELP zeno_component_queue_size Size of component queues in the processing pipeline
# TYPE zeno_component_queue_size gauge
zeno_component_queue_size{component="reactor_to_preprocessor",hostname="host",project="proj",version="ver"} 0
zeno_component_queue_size{component="preprocessor_to_archiver",hostname="host",project="proj",version="ver"} 2
zeno_component_queue_size{component="archiver_to_postprocessor",hostname="host",project="proj",version="ver"} 1
...

TUI Integration

Queue sizes are now displayed in the terminal interface as part of the stats panel, updating in real-time every 500ms alongside existing metrics.

Usage Examples

# Start crawl with Prometheus metrics
./Zeno get url --api --prometheus --api-port 9000 https://example.com

# Query component queue sizes
curl http://localhost:9000/metrics | grep component_queue

# Use with Prometheus/Grafana for monitoring and alerting
# Use TUI for live monitoring
./Zeno get url --tui https://example.com

Technical Implementation

• Thread-safe: Uses RWMutex for concurrent access to channel tracking
• Zero overhead: Minimal performance impact when not actively monitoring
• Dependency injection: Testable design with proper separation of concerns
• Real-time: Queue sizes calculated using len(channel) for instant feedback
• Prometheus integration: Updates every 250ms alongside other stats

Benefits

1. Bottleneck Identification: Quickly identify which components are slower than others
2. Performance Optimization: Data-driven capacity planning and tuning
3. Monitoring Integration: Native Prometheus metrics enable integration with standard monitoring tools
4. Historical Analysis: Time-series data for trend analysis and alerting

This implementation provides pipeline visibility for operators to identify and resolve performance bottlenecks effectively using standard monitoring infrastructure.

Fixes #471.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.